Bayesian variable order Markov models

We present a simple, effective generalisation of variable order Markov models to full online Bayesian estimation. The mechanism used is close to that employed in context tree weighting. The main contribution is the addition of a prior, conditioned on context, on the Markov order. The resulting construction uses a simple recursion and can be updated efficiently. This allows the model to make predictions using more complex contexts, as more data is acquired, if necessary. In addition, our model can be alternatively seen as a mixture of tree experts. Experimental results show that the predictive model exhibits consistently good performance in a variety of domains. We consider Bayesian estimation of variable order Markov models (see Begleiter et al., 2004, for an overview). Such models create a tree of partitions, where the disjoint sets of every partition correspond to different contexts. We can associate a sub-model or expert with each context in order to make predictions. The main contribution of this paper is a conditional prior on the Markov order—or equivalently the context depth. This is based on a recursive construction that estimates, for each context at a certain depth k, whether it makes better predictions than the predictions of contexts at depths smaller than k. This simple model defines a mixture of variable order Marko models and its parameters can be updated in closed form in time O (D) for trees of depth D with each new observation. For unbounded length contexts, the complexity of the algorithm is O ( T 2 ) for an input sequence of length T . Furthermore, it exhibits robust performance in a variety of tasks. Finally, the model is easily extensible to controlled processes. Appearing in Proceedings of the 13 International Conference on Artificial Intelligence and Statistics (AISTATS) 2010, Chia Laguna Resort, Sardinia, Italy. Volume 9 of JMLR: W&CP 9. Copyright 2010 by the authors. The following section presents our setup for the sequence prediction problem and introduces notation. Section 2 defines the proposed Bayesian variable order Markov model (henceforth BVMM) and derives a closed form sequential update for the model parameters. Section 3 gives a brief overview of other tree based models, as well as a standard Markov chain mixture, and discusses their relation to BVMMs. Experimental comparisons between various models are given in Section 4, on a number of sequential prediction problems. The paper concludes with Section 5, which also discusses extensions and applications to reinforcement learning.